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Gauging  Future  Demand  for  Veterans’  Health  Care 
Does  the  VA  Have  the  Forecasting  Tools  It  Needs T2 

Before  the  Committee  on  Veterans’  Affairs 
United  States  House  of  Representatives 

April  29,  2009 

Mr.  Chairman  and  distinguished  members  of  the  Committee,  thank  you  for  inviting  me  to  testify 
today.  It  is  an  honor  and  pleasure  to  be  here.  I  will  discuss  the  findings  from  RAND’s  recent 
evaluation  of  the  VA’s  Enrollee  Healthcare  Project  Model  as  it  relates  to  the  topic  of  your  hearing 
today.  More  specifically,  my  testimony  will  briefly  review  the  findings  from  our  evaluation,  discuss 
the  model’s  utility  to  support  the  proposed  advance  appropriation  of  the  VA  budget,  and  discuss 
recommendations  for  improving  the  model. 

Background 

In  1996,  the  mission  of  the  Veterans  Administration  (VA)  broadened  dramatically.  The  Veterans’ 
Health  Care  Eligibility  Reform  Act  of  1996  transformed  the  VA  from  an  episodic  provider  of 
inpatient  care  for  veterans  to  a  comprehensive  health  care  provider  responsible  for  all  the  medical 
needs  of  veterans  who  enroll.  To  support  budgeting  and  planning  for  this  broader  mission,  the  VA 
relies  on  a  complex  model  known  as  the  Enrollee  Health  Care  Projection  Model  (EHCPM).  This 
model  predicts  future  demand  for  veterans’  health  care  needs.  The  VA  asked  RAND  (in 
conjunction  with  an  independent  actuary)  to  evaluate  the  model,  which  was  developed  and  is 
operated  by  an  actuarial  consulting  firm. 


The  RAND  team  reviewed  how  the  model  works  and  addressed  three  main  questions  in  its 
evaluation: 

•  Does  the  modeling  approach  support  long-term  budget  planning  and  policy  analysis? 

•  Does  it  accurately  project  VA  service  demand  and  costs? 

•  Is  the  design  and  operation  of  the  model  transparent  to  users  and  outside  parties? 


1  The  opinions  and  conclusions  expressed  in  this  testimony  are  the  author's  alone  and  should  not  be 
interpreted  as  representing  those  of  RAND  or  any  of  the  sponsors  of  its  research.  This  product  is  part  of  the 
RAND  Corporation  testimony  series.  RAND  testimonies  record  testimony  presented  by  RAND  associates  to 
federal,  state,  or  local  legislative  committees;  government-appointed  commissions  and  panels;  and  private 
review  and  oversight  bodies.  The  RAND  Corporation  is  a  nonprofit  research  organization  providing  objective 
analysis  and  effective  solutions  that  address  the  challenges  facing  the  public  and  private  sectors  around  the 
world.  RAND’s  publications  do  not  necessarily  reflect  the  opinions  of  its  research  clients  and  sponsors. 

2  This  testimony  is  available  for  free  download  at  http://www.rand.org/pubs/testimonies/CT327/. 
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Overall,  RAND’s  evaluation  found  that  the  EHCPM  is  useful  for  short-term  budget  planning,  but  is 
less  useful  for  longer-range  planning,  especially  in  a  dynamic  policy  environment.  Fortunately,  the 
model  is  structured  in  a  way  that  would  allow  modifications  to  support  longer-term  policy  and 
planning  applications  without  disrupting  its  usefulness  for  near-term  budget  planning. 

How  Does  the  Model  Work? 

The  EHCPM  estimates  the  use  of  VA  services  in  a  base  year  for  each  service  category  (e.g., 
inpatient  care,  office  visits),  using  proprietary  benchmarks  derived  from  utilization  in  commercial 
health  plans.  The  costs  associated  with  the  estimated  use  of  each  service  are  derived  from  data 
provided  by  the  VA’s  cost  accounting  system.  In  the  next  step,  the  EHCPM  estimates  budget- 
year  service  use  and  the  unit  cost  of  services.  These  estimates  are  based  on  anticipated  changes 
in  demand  for  VA  care,  the  efficiency  and  intensity  of  care  provided  by  the  VA  system,  and  overall 
projected  medical  inflation  in  the  United  States.  In  any  given  year,  the  VA  forecasts  expenditures 
for  each  service  by  multiplying  expected  enrollment,  forecast  utilization,  and  forecast  unit  costs. 

Does  the  Model  Support  Budgeting  and  Policy  Analysis? 

The  RAND  evaluation  found  that  the  EHCPM  supports  VA’s  short-term  budget  planning  and 
monitoring  in  a  stable  policy  and  practice  environment.  The  model  identifies  factors  that  drive 
specific  types  of  spending  or  spending  for  specific  types  of  enrollees  and  can  adjust  those  factors 
as  needed.  Model  results  can  also  help  the  VA  to  develop  more  informed  strategies  for  managing 
expenditures.  In  addition,  the  current  model  allows  the  VA  to  monitor  budget  execution  and 
performance  relative  to  pre-established  benchmarks.  Assuming  there  are  no  short-term  “shocks” 
to  the  system,  only  the  accuracy  and  timeliness  of  VA  data  systems — not  the  model’s  structure — 
limit  the  EHCPM’s  utility  for  short-term  budget  planning  and  monitoring. 

However,  for  longer-term  strategic  planning  and  policy  analysis,  the  model  could  yield  misleading 
results  because  the  model  structure  does  not  account  for  two  things:  key  drivers  of  future  demand 
for  VA  care  and  the  costs  of  delivering  it.  Using  the  model  to  inform  scenarios  beyond  the  current 
policy  and  budgetary  environment  requires  information  about  a  wide  range  of  factors,  including 
the  VA’s  future  cost  structure,  how  rapidly  the  VA  can  expand  its  capacity  to  meet  demand, 
factors  driving  enrollment,  and  the  relationships  among  enrollee  health  status,  VA  treatment 
capacity,  and  enrollees’  preferences  for  treatment  in  VA  facilities  versus  other  facilities.  In  many 
cases,  required  information  does  not  exist  or  was  not  available  to  model  developers.  In  the 
absence  of  such  information,  model  forecasts  rely  on  a  number  of  unrealistic  assumptions.  Thus, 
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substantial  modifications  to  model  subcomponents  and  enhancements  of  supporting  data  inputs 
would  likely  be  required  before  the  EHCPM  could  effectively  support  longer-range  planning. 

Is  the  Model  Accurate? 

The  model’s  ability  to  accurately  predict  the  level  of  resources  needed  by  the  VA  in  future  years 
to  meet  projected  demand  is  uncertain.  The  discretionary  nature  of  the  VA’s  budget  complicates 
the  comparison  between  model  projections  and  actual  expenditures.  Under  a  discretionary 
budget,  the  VA  does  not  have  the  authority  to  spend  more  than  Congress  appropriates.  If 
demand  for  VA  services  cannot  be  satisfied  under  its  appropriation,  then  actual  expenditures  will 
reflect  the  constraints  inherent  in  the  appropriation  and  not  actual  demand  for  VA  services. 

Model  accuracy  becomes  less  certain  as  it  is  used  to  project  the  impact  of  policy  and  budget 
scenarios  farther  from  the  status  quo.  The  main  source  of  this  uncertainty  stems  from  the  fact  that 
the  EHCPM  begins  its  expenditure  projection  with  the  VA’s  congressional  budget  allocation, 
rather  than  with  an  independent  measure  of  resource  needs.  Past  VA  budgets  are  imbedded  in 
expenditure  projections  through  the  derivation  of  the  model’s  unit  cost  measure  and  through  the 
calibration  of  utilization  benchmarks  to  actual  VA  workload  data.  In  other  words,  the  accuracy  of 
the  model  is  uncertain  because  there  exists  no  expenditure  information  independent  of  the  VA 
appropriation  with  which  to  formulate  a  “gold”  standard  against  which  to  compare  model 
projections. 

Is  the  Model  Transparent? 

It  is  important  that  large,  complex  policy  models  like  the  EHCPM  be  transparent.  A  lack  of 
transparency  can  undermine  the  credibility  of  the  model  and  make  the  model  difficult  to  operate 
and  manage.  The  overall  structure  of  the  model  is  relatively  easy  for  users  and  outside  evaluators 
to  understand.  However,  the  model’s  subcomponents  are  less  transparent.  Transparency  of  the 
model’s  subcomponents  is  limited  by  several  factors:  complicated  algorithms  that  are  used  to  set 
parameters  of  model  subcomponents;  uneven  and  often  incomplete  model  documentation; 
reliance  on  data  and  clinical  efficiency  benchmarks  that  are  proprietary  to  the  contractor  who 
operates  EHCPM  and  therefore  not  available  for  outside  review;  and  the  lack  of  a  standing 
process  for  obtaining  independent  review. 
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Does  the  Model  Support  Advanced  Appropriation? 


If  enacted,  the  Veterans  Health  Care  Budget  Reform  and  Transparency  Act  of  2009  (HR1 016) 
would  give  Congress  the  ability  to  appropriate  funds.  Advance  appropriation  would,  in  essence, 
lengthen  the  time  horizon  over  which  the  model  forecasts  resource  requirements  from  three  years 
in  the  current  model  baseline  to  four  years.  Under  the  current  system,  for  example,  the  VA  plans 
the  FY  201 2  budget  request  using  a  version  of  the  model  with  an  FY  2009  baseline.  Under 
advanced  appropriations,  the  FY  2009  baseline  would  inform  the  FY  2013  budget  request. 
Generally  the  farther  out  the  forecast,  the  less  accurate  the  projections. 

Advance  appropriations  may  serve  to  mitigate  the  challenges  of  operating  a  large,  complex  health 
care  system  posed  by  delayed  enactment  of  the  VA’s  annual  budget.  At  the  same  time,  the 
expanded  time  period  between  budget  appropriation  and  the  time  spending  actually  occurs 
makes  it  even  more  imperative  that  the  VA  have  robust  budget  planning  tools  at  its  disposal. 

Again,  our  findings  suggest  that  the  model  is  useful  for  short-term  budget  planning  to  the  extent 
that  the  VA’s  treatment  capacity  and  the  policy  environment  surrounding  the  VA  remain  stable. 
This  is  because  model  projections  are  tied  to  past  VA  budgets  and  not  an  independent  measure 
of  resource  requirements.  The  longer  the  period  of  time  between  the  baseline  year  and  the 
budget  planning  year,  the  higher  the  risk  that  that  past  budgets  do  not  reflect  the  resources 
required  by  the  VA  to  achieve  its  mission.  Both  the  conflicts  in  Iraq  and  Afghanistan  and  the 
impact  of  the  current  recession  on  the  employment  and  private  health  insurance  coverage  of 
veterans  raise  concerns  about  the  impact  of  a  changing  policy  environment  for  the  robustness  of 
short-term  model  forecasts.  Lengthening  the  forecasted  time  period  under  advanced 
appropriation  amplifies  these  concerns. 

Recommendations  for  Improving  the  EHCPM 

Based  on  the  results  of  our  evaluation,  we  recommend  that  VA  take  a  number  of  steps  to 
increase  the  model’s  ability  to  generate  budget  forecasts  that  are  robust  to  changes  in  the  policy 
environment  over  longer  periods  of  time. 

Develop  a  Methodology  for  Estimating  Demand-Based  Resource  Requirements 

We  recommend  that  the  VA  develop  and  apply  a  method  to  enhance  the  model’s  capacity  to 
estimate  resource  requirements  that  reflect  any  unmet  demand  using  VA  data  sources.  Budget 
forecasts  are  not  fully  demand-based,  because  calibrating  commercial  utilization  benchmarks  to 
VA  workload  data  imbeds  constraints  that  arise  from  VA  capacity  constraints  in  the  baseline 
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utilization  estimates.  Forecasting  of  resource  requirements  requires  measures  of  demand  that  are 
responsive  to  changes  in  VA  treatment  capacity,  benefit  generosity,  and  case-mix.  Estimating 
demand  for  VA  health  care  for  these  purposes  requires  the  development  and  application  of 
methodologies  for  (1)  estimating  the  utilization  that  would  have  occurred  in  the  absence  of 
constraints  on  VA’s  capacity  to  deliver  care,  (2)  estimating  the  relationship  between  VA  benefit 
generosity  relative  to  other  payors  and  demand  for  VA  care,  and  (3)  estimating  the  relationship 
between  enrollee  health  status  and  demand  for  VA  care. 

These  methodologies  could  be  developed  by  combining  VA  workload  data  with  data  describing 
treatment  capacity  and  various  sources  of  data  on  enrollee  reliance.  Exploiting  variation  in  VA 
capacity  across  locations  and  over  time  could  allow  modelers  to  infer  demand  for  VA  care  in 
constrained  markets  from  administrative  workload  data  collected  from  unconstrained  regions  and 
time  periods,  controlling  for  case-mix.  The  ability  to  control  for  and  measure  enrollee’s  partial 
reliance  on  VA  care  will  require  additional  data  beyond  VA  workload  and  VA  treatment  capacity. 
As  reflected  in  the  current  model,  such  information  is  likely  to  include  Medicare  claims  data  linked 
to  VA  workload  and  self-reported  reliance  from  survey  data. 

To  assure  full  exploration  of  the  capabilities  and  limitations  of  VA  administrative  and  survey  data 
sources  in  estimating  unconstrained  demand  for  VA  health  care,  we  recommend  the  VA  consult 
with  a  wide  variety  of  independent  experts  including  actuaries,  economists,  and  in  particular, 
individuals  with  experience  aggregating  VISN-level  workload  data  to  conduct  national-level 
analyses. 

Use  Survey-Based  Methods  to  Strengthen  Demand  Forecasting  and  Policy  Analysis 

We  recommend  the  VA  use  survey-based  methods  to  strengthen  forecasting  and  policy  analysis 
capabilities.  The  fact  that  veterans  do  not  receive  medical  care  exclusively  from  the  VA  makes  it 
impossible  to  project  future  demand  for  VA  health  care  from  administrative  data  alone.  For 
example,  VA  eligibility  data  does  not  contain  information  needed  to  measure  the  effect  of 
changes  in  availability  and  generosity  of  employer-sponsored  health  insurance  benefits  on 
demand  for  enrollment  and  use  of  VA  health  care  services.  Likewise,  it  is  not  possible  to 
distinguish  the  effect  of  reliance  from  veteran  health  status  when  using  VA  workload  data  to 
predict  future  demand. 

The  current  survey  of  enrollees  provides  useful  information  in  estimating  demand  for  VA  care  by 
asking  insurance  status  and  source,  anticipated  use  of  VA  health  care,  health  and  functional 
status,  and  use  of  VA  and  non-VA  health  care.  However,  the  utility  of  the  current  survey  could  be 
greatly  increased  if  the  sample  (for  both  respondents  and  non  respondents)  were  designed  to  be 
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linkable  to  VA  workload  data,  included  non  enrollees,  and  was  stratified  to  ensure  representation 
of  veterans  across  VA  markets  identified  as  being  supply  constrained  or  having  excess  capacity. 
Likewise,  the  utility  of  the  survey  could  be  greatly  increased  if  the  questionnaire  were  modified  to 
include  screening  questions  regarding  diagnosed  health  conditions,  utilization  of  services  in  broad 
service  categories,  and  more  information  about  other  health  insurance  coverage  availability  and 
costs. 

We  recommend  that  the  VA  consult  a  variety  of  sampling  statisticians  and  survey  design  experts 
in  making  design  changes  to  assure  that  modifications  support  to  the  greatest  extent  possible 
VA’s  objectives  related  to  forecasting  and  policy  analysis  while  minimizing  respondent  burden 
and  cost  to  the  VA. 

Explore  the  Utility  and  Feasibility  of  Improving  Unit  Cost  Measures  Through  Alternative 
Approaches 

We  recommend  the  VA  consult  with  a  variety  of  experts  to  improve  its  understanding  of  the  likely 
biases  resulting  from  the  current  costing  methods,  whether  and  how  alternative  approaches  could 
improve  unit  cost  estimates.  We  found  that  the  method  used  to  derive  unit  costs  has  the  potential 
to  produce  biased  expenditure  projections.  The  potential  for  bias  stems  from  the  implicit 
assumption  that  per  unit  costs  do  not  vary  with  changes  over  time  in  the  number  of  treated 
patients.  In  essence,  the  model  assumes  that  VA  pays  for  care  on  a  fee-for-service  basis,  similar 
to  Medicare.  Our  analyses  suggest  that  the  potential  for  bias  is  greatest  for  services  with  large 
fixed  cost  components  for  both  capacity  constrained  markets  and  markets  with  substantial  excess 
capacity. 

Alternative  approaches  may  yield  more  valid  and  accurate  expenditure  projections  that  can  be 
more  readily  related  to  the  VA’s  actual  expenditures.  In  particular,  we  recommend  the  VA  explore 
whether  it  is  feasible  to  implement  a  staffing  model  using  VA’s  cost  accounting  system.  A  staffing 
model  explicitly  maps  resources  expended  in  a  delivery  system  to  anticipated  demand  based  on 
cost  histories  of  service  for  major  expenditure  components,  such  as  diagnostic  equipment,  office 
supplies,  purchased  services,  administration,  salaries  and  benefits  and  rent. 

We  recommend  that  the  VA  consult  actuaries,  economists  with  expertise  in  costing  methods,  and 
individuals  familiar  with  VA  data  systems  to  recommend  a  strategy  for  analyzing  the  problems 
associated  with  the  current  costing  method  and  to  assess  whether  a  staffing  model  (or  alternative 
costing  method)  is  likely  to  result  in  improved  accuracy  and  could  be  supported  using  the  VA’s 
current  cost  accounting  system. 


6 


The  implementation  of  a  staffing  model  as  a  basis  for  forecasting  VA  resource  requirements 
would  be  time-consuming  and  resource  intensive.  However,  investing  in  the  capacity  to  develop, 
implement,  and  maintain  a  staffing  model  would  most  likely  produce  returns  beyond  the  ability  to 
improve  the  quality  of  model-based  expenditure  projections.  In  particular,  the  development  of  a 
staffing  model  would  inform  the  development  and  refinement  of  productivity  benchmarks  for 
physicians,  physician  support  staff,  and  medical  equipment  and  the  accurate  measurement  of 
performance  relative  to  these  benchmarks.  A  staffing  model  can  also  help  the  VA  to  evaluate 
potential  return  from  investments  in  cost  saving  or  quality  enhancing  technology. 

Consider  Streamlining  the  Current  Model  for  Short-Term  Budget  Planning 

If  model  enhancements  required  to  improve  the  model’s  capability  to  support  long-term  planning 
and  analysis  prove  impractical,  we  recommend  that  the  VA  streamline  the  current  model  to 
provide  more  transparent  support  for  short-term  budget  planning.  Streamlining  would  entail 
discontinued  use  of  commercial  utilization  benchmarks,  the  development  of  VA-specific  utilization 
benchmarks,  and  the  simplification  of  trend  assumptions  used  to  project  base  year  utilization 
forward  3  years.  We  expect  a  streamlined  model  based  on  VA  data  would  be  close  in  structure  to 
the  current  methodology  used  to  project  expenditures  for  non  modeled  services  (e.g.,  outpatient 
mental  health  services,  over-the-counter  drugs  and  supplies).  We  expect  that  commercial 
benchmarks  will  prove  useful  in  isolated  instances  in  which  VA  data  systems  do  not  adequately 
capture  utilization  of  covered  services. 

Because  VA  workload  drives  short-term  expenditure  projections  under  the  current  model  through 
the  calibration  of  estimated  utilization  to  actual  utilization  using  VA  workload  data,  discontinuing 
use  of  commercial  utilization  benchmarks  will  substantially  reduce  complexity  and  increase 
transparency  without  substantially  affecting  the  continuity  of  the  VA’s  budget  planning  process. 

The  VA  is  substantially  larger  than  many  large  health  insurers  who  use  their  own  experience  for 
budgeting  and  strategic  planning  purposes.  For  this  reason,  it  should  be  feasible  to  use  standard 
statistical  methods  and  the  aggregation  of  data  across  multiple  time  periods  to  develop 
assumptions  regarding  variation  in  VA  utilization  by  age,  priority-level,  and  geographic  region, 
even  when  the  volume  of  workload  is  low  for  a  given  service. 

Use  a  Wide  Range  of  Expertise  to  Enhance  Validity,  Accuracy,  and  Credibility 

We  recommend  that  the  VA  draw  on  a  broader  range  of  expertise  than  is  currently  being 
employed  for  the  purpose  of  enhancing  the  validity,  accuracy,  and  external  credibility  of  the  model. 
Our  evaluation  suggested  that  model  development  activities  were  staffed  solely  by  actuaries  with 
support  from  programmers  with  limited  support  from  outside  experts.  However,  many  modeling 
tasks  are  well  within  the  purview  of  other  disciplines,  including  economics,  statistics,  health 
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services  research,  and  epidemiology.  Many  individuals  with  backgrounds  in  these  areas  have 
relevant  modeling  experience  and  expertise  in  specialized  analytic  approaches  needed  to 
address  model  limitations  identified  in  our  evaluation.  These  approaches  include  cost 
measurement,  estimating  demand  in  supply  constrained  environments,  and  case-mix  adjustment 
using  administrative  data. 

Initiate  Periodic  External  Review  of  the  Model 

We  recommend  that  VA  initiate  periodic  review  of  the  model  by  independent  experts  recruited 
from  outside  the  VA.  Independent  review  helps  to  insure  model  credibility  in  the  eyes  of 
stakeholders  who  may  not  have  the  time  or  expertise  to  evaluate  the  model  themselves.  To  our 
knowledge,  the  EHCPM  model  has  not  been  subject  to  external  review  prior  to  our  evaluation. 
Sponsors  of  other  large  scale  forecasting  models,  such  as  the  models  used  by  the  Social  Security 
Administration  and  the  Center  for  Medicare  and  Medicaid  Services  (CMS),  periodically  engage 
panels  of  experts  to  review  modeling  methodologies,  key  assumptions,  and  model  outputs. 
Proceedings  from  these  meetings  could  serve  as  models  in  establishing  a  review  process. 

Involve  Technical  Writers  in  Documentation  Process 

We  recommend  that  the  VA  increase  transparency  and  credibility  through  the  use  of  technical 
writers  to  improve  the  quality  of  model  documentation.  As  we  note  earlier  in  this  report,  any  valid 
approach  to  projecting  future  VA  health  care  expenditures  under  enrollment  reform  policies  is 
likely  to  involve  a  very  high  degree  of  complexity.  Given  this  complexity,  it  is  crucial  that  model 
documentation  be  comprehensive,  be  clear,  and  meet  the  reviewers’  expectations  with  respect  to 
the  appropriate  level  of  detail.  Technical  writers  have  the  skills  and  experience  to  assure  that 
these  goals  are  met  through  the  use  of  unambiguous  language  and  visual  formatting. 

Capture  Institutional  Knowledge  through  the  Addition  of  Internal  Analytic  Staff 

We  recommend  that  the  VA  add  internal  analytic  staff  to  participate  in  model  development  and 
related  activities  in  order  to  accelerate  institutional  learning  and  increase  the  return  on  the  VA’s 
investment  in  the  model.  Our  evaluation  did  not  support  conclusions  one  way  or  the  other  about 
the  desirability  of  outsourcing  model  development  and  related  activities.  Our  evaluation  did, 
however,  raise  concerns  about  outsourcing  the  institutional  knowledge  that  arises  through  day-to- 
day  participation  in  model-related  activities  and  interaction  with  other  VA  staff,  both  formal  and 
informal.  In  our  view,  the  capture  of  institutional  knowledge  is  key  to  enhancing  the  VA’s  return  on 
its  investment  in  the  model.  Internal  analytic  staff  would  likely  be  familiar  with  the  VA’s  strategic 
mission  and  have  detailed  knowledge  of  VA  data  systems.  Thus,  in  addition  to  the  general 
knowledge  enhancement  and  related  benefits  achieved  by  the  analytic  staff,  such  individuals 
could  also  help  to  enhance  the  strategic  value  of  the  VA  data  systems. 
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